Skip to content

PYTHON-3186 Avoid SDAM heartbeat timeouts on AWS Lambda #912

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 10 commits into from
Mar 30, 2022
Merged

PYTHON-3186 Avoid SDAM heartbeat timeouts on AWS Lambda #912

merged 10 commits into from
Mar 30, 2022
+148 −5

Conversation

ShaneHarvey
Copy link
Member

@ShaneHarvey ShaneHarvey commented Mar 29, 2022 •
edited
Loading

Poll monitor socket with timeout=0 one last time after timeout expires.

https://jira.mongodb.org/browse/PYTHON-3186

@ShaneHarvey ShaneHarvey marked this pull request as ready for review March 30, 2022 01:42
blink1073
blink1073 approved these changes Mar 30, 2022
View reviewed changes
Copy link
Member

@blink1073 blink1073 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@ShaneHarvey
Copy link
Member Author

Oof, the test fails on macOS in a similar way to Windows. I'm going to try and rewrite the test to use subprocess.Popen instead of multiprocessing.

@ShaneHarvey
Copy link
Member Author

ShaneHarvey commented Mar 30, 2022 •
edited
Loading

Updated the test to use Popen which should hopefully work across all platforms. Without the changes to network.py this test fails, like this:

FAIL: test_sigstop_sigcont (test.test_client.TestClient)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/shane/git/mongo-python-driver/test/test_client.py", line 1724, in test_sigstop_sigcont
    self.assertNotIn("ServerHeartbeatFailedEvent", log_output)
AssertionError: 'ServerHeartbeatFailedEvent' unexpectedly found in "2022-03-30 12:52:10,314 INFO <ServerHeartbeatStartedEvent ('localhost', 27018)>\n2022-03-30 12:52:10,316 INFO <ServerHeartbeatSucceededEvent ('localhost', 27018) duration: 0.0005846569547429681, awaited: False, reply: <pymongo.hello.Hello object at 0x7f7c2e628778>>\n2022-03-30 12:52:10,316 INFO <ServerHeartbeatStartedEvent ('localhost', 27017)>\n2022-03-30 12:52:10,317 INFO <ServerHeartbeatStartedEvent ('localhost', 27019)>\n2022-03-30 12:52:10,317 INFO <ServerHeartbeatStartedEvent ('localhost', 27018)>\n2022-03-30 12:52:10,319 INFO <ServerHeartbeatSucceededEvent ('localhost', 27017) duration: 0.0010778519790619612, awaited: False, reply: <pymongo.hello.Hello object at 0x7f7c2e63ca98>>\n2022-03-30 12:52:10,319 INFO <ServerHeartbeatSucceededEvent ('localhost', 27019) duration: 0.000964289007242769, awaited: False, reply: <pymongo.hello.Hello object at 0x7f7c2e63cb38>>\n2022-03-30 12:52:10,320 INFO TEST STARTED\n2022-03-30 12:52:10,320 INFO <ServerHeartbeatStartedEvent ('localhost', 27019)>\n2022-03-30 12:52:10,320 INFO <ServerHeartbeatStartedEvent ('localhost', 27017)>\n2022-03-30 12:52:10,823 INFO <ServerHeartbeatSucceededEvent ('localhost', 27018) duration: 0.5050969959702343, awaited: True, reply: <pymongo.hello.Hello object at 0x7f7c2e63c9f8>>\n2022-03-30 12:52:10,823 INFO <ServerHeartbeatStartedEvent ('localhost', 27018)>\n2022-03-30 12:52:10,825 INFO <ServerHeartbeatSucceededEvent ('localhost', 27019) duration: 0.5044320530141704, awaited: True, reply: <pymongo.hello.Hello object at 0x7f7c2e63cb38>>\n2022-03-30 12:52:10,825 INFO <ServerHeartbeatStartedEvent ('localhost', 27019)>\n2022-03-30 12:52:10,825 INFO <ServerHeartbeatSucceededEvent ('localhost', 27017) duration: 0.5047361569595523, awaited: True, reply: <pymongo.hello.Hello object at 0x7f7c2e63cc78>>\n2022-03-30 12:52:10,826 INFO <ServerHeartbeatStartedEvent ('localhost', 27017)>\n2022-03-30 12:52:13,172 WARNING <ServerHeartbeatFailedEvent ('localhost', 27017) duration: 2.3468628809787333, awaited: {'processId': ObjectId('623ba99b307680f47072b66b'), 'counter': 167}, reply: NetworkTimeout('localhost:27017: timed out',)>\n2022-03-30 12:52:13,173 INFO <ServerHeartbeatStartedEvent ('localhost', 27017)>\n2022-03-30 12:52:13,174 WARNING <ServerHeartbeatFailedEvent ('localhost', 27018) duration: 2.35050753201358, awaited: {'processId': ObjectId('623ba99d6151e1c236bf8fb8'), 'counter': 155}, reply: NetworkTimeout('localhost:27018: timed out',)>\n2022-03-30 12:52:13,174 INFO <ServerHeartbeatStartedEvent ('localhost', 27018)>\n2022-03-30 12:52:13,174 WARNING <ServerHeartbeatFailedEvent ('localhost', 27019) duration: 2.349525861965958, awaited: {'processId': ObjectId('623ba99fb7b95b1439017c72'), 'counter': 66}, reply: NetworkTimeout('localhost:27019: timed out',)>\n2022-03-30 12:52:13,175 INFO <ServerHeartbeatStartedEvent ('localhost', 27019)>\n2022-03-30 12:52:13,177 INFO <ServerHeartbeatSucceededEvent ('localhost', 27017) duration: 0.001169831957668066, awaited: False, reply: <pymongo.hello.Hello object at 0x7f7c2e65f228>>\n2022-03-30 12:52:13,178 INFO <ServerHeartbeatSucceededEvent ('localhost', 27018) duration: 0.0011216229759156704, awaited: False, reply: <pymongo.hello.Hello object at 0x7f7c2e65f318>>\n2022-03-30 12:52:13,178 INFO <ServerHeartbeatStartedEvent ('localhost', 27017)>\n2022-03-30 12:52:13,178 INFO <ServerHeartbeatStartedEvent ('localhost', 27018)>\n2022-03-30 12:52:13,178 INFO <ServerHeartbeatSucceededEvent ('localhost', 27019) duration: 0.001337455993052572, awaited: False, reply: <pymongo.hello.Hello object at 0x7f7c2e63c908>>\n2022-03-30 12:52:13,179 INFO <ServerHeartbeatStartedEvent ('localhost', 27019)>\n2022-03-30 12:52:13,679 INFO TEST COMPLETED\n"

@ShaneHarvey
Copy link
Member Author

Tests should finally be passing now, besides the link checker which just started failing.

blink1073
blink1073 approved these changes Mar 30, 2022
View reviewed changes
Copy link
Member

@blink1073 blink1073 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@ShaneHarvey ShaneHarvey merged commit c58950a into mongodb:master Mar 30, 2022
@ShaneHarvey ShaneHarvey deleted the PYTHON-3186 branch March 30, 2022 21:32
juliusgeo pushed a commit to juliusgeo/mongo-python-driver that referenced this pull request Apr 7, 2022
@ShaneHarvey @juliusgeo
PYTHON-3186 Avoid SDAM heartbeat timeouts on AWS Lambda (mongodb#912)
c3ad9aa 
Poll monitor socket with timeout=0 one last time after timeout expires.
This avoids heartbeat timeouts and connection churn on Lambda and other FaaS envs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@blink1073 blink1073 blink1073 approved these changes

@juliusgeo juliusgeo Awaiting requested review from juliusgeo

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ShaneHarvey @blink1073